System and method for calculating a score for a chain of interactions in a call center

ABSTRACT

Calculating a score for a chain of interactions in a call center may include: during a first training phase, train a first model which, given an interaction and interaction metadata, predict an initial estimated customer satisfaction score; during a second training phase, train a second model which, given an interaction and interaction metadata, text and metadata of an immediately preceding interaction in a chain of interactions, and features of the chain, predict a refined estimated customer satisfaction score; and during an inference phase: given a chain of interactions and metadata of each interaction, compute an initial estimated customer satisfaction score for each interaction using the first model; beginning with a second interaction in the chain and metadata of each interaction, compute a refined estimated customer satisfaction score for each interaction using the second model; combine the interaction scores into a combined customer satisfaction score; and output the score.

FIELD OF THE INVENTION

The present invention is in the field of customer experience analytics.In particular, the present invention is directed towards systems andmethods for calculating a score for a chain of interactions in a callcenter.

BACKGROUND OF THE INVENTION

Currently, call center analytics systems and methods apply customerexperience metrics to voice interactions in order to evaluate agents,rank business processes, and/or to find examples of particularly badinteractions. For example, one customer experience model, called CXMetric, is a model that is trained on interaction content (e.g., text)and uses customer surveys as labels. Interaction chains are series ofinteractions performed by the same customer, usually for the same orrelated reasons, during a relatively short period of time. Currentlyavailable analytics systems and methods evaluate each interactioncomprising the chain independently. However, this method producesinaccurate results, as customer experience also depends on the previousinteraction or interactions in the chain. Furthermore, the number ofinteraction chains in which every interaction has a follow-up survey(from the interactant) associated with it is typically very small, andaccordingly does not provide enough data for training such models.

What are needed, therefore, are systems and methods which evaluate eachinteraction in a series or chain of interactions while taking intoaccount the evolution of the customer experience over the chain.

SUMMARY OF EMBODIMENTS OF THE INVENTION

Various embodiments of the invention include systems and methods ofcalculating a score for a chain of interactions in a call center,performed on a computer having a processor, memory, and one or more codesets stored in the memory and executed by the processor, the systems andmethods including: during a first training phase, training a first modelwhich, given a processed text of a given interaction and metadataassociated with the given interaction, predicts an initial estimatedcustomer satisfaction score for the given interaction; during a secondtraining phase, training a second model which, given a processed text ofa given interaction and metadata associated with the given interaction,processed text and associated metadata of an immediately precedinginteraction in a given chain of interactions, and one or more featuresof the chain of interactions, predicts a refined estimated customersatisfaction score for the given interaction; and during an inferencephase: given a chain of interactions and associated metadata of eachinteraction, computing an initial estimated customer satisfaction scorefor each interaction in the chain of interactions using the first model;beginning with a second interaction in the given chain of interactionsand associated metadata of each interaction, computing a refinedestimated customer satisfaction score for each interaction in the chainof interactions using the second model; combining the refined estimatedcustomer satisfaction score of each interaction into a combined customersatisfaction score for the given chain of interactions; and outputtingthe combined customer satisfaction score.

In some embodiments, the first training phase may include: collecting bythe processor a plurality of interactions, in which each of theplurality of interactions comprises associated survey information;converting by the processor any non-text interactions of the pluralityof interactions into text; preprocessing by the processor text of eachof the plurality of interactions; extracting by the processor aplurality of metadata features from each of the plurality ofinteractions; and training by the processor the first model based on thetext, metadata, and survey information of each of the plurality ofinteractions.

In some embodiments, the second training phase may include: for the sameplurality of interactions, as well as an immediately precedinginteraction of each of the plurality of interactions in a given chain ofinteractions: converting by the processor any non-text interactions intotext; preprocessing by the processor text of each of the plurality ofinteractions and immediately preceding interactions; extracting by theprocessor a plurality of metadata features from each of the plurality ofinteractions and immediately preceding interactions; applying by theprocessor the first model to each of the plurality of interactions andimmediately preceding interactions; generating by the processor aninterim estimated customer satisfaction score for each of the pluralityof interactions and immediately preceding interactions; and training bythe processor the second model based on one or more features of thechain of interactions, and the interim estimated customer satisfactionscore for each of the plurality of interactions and immediatelypreceding interactions.

In some embodiments, an interaction may include at least one of a voicerecording, an e-mail, and a messaging chat. In some embodiments, theassociated survey information may include a survey score provided by acustomer who was a party to the given interaction. In some embodiments,preprocessing may include transforming by the processor at least one ofone or more words and one or more phrases from the text of a giveninteraction into one or more word embedding vectors, using one or morepre-trained in-domain embeddings.

In some embodiments, one or more of the first model and the second modelmay include a Convolutional Neural Network architecture (CNN) and a DeepNeural Network architecture (DNN), and wherein the CNN and the DNN areload-balanced with an auxiliary output of the CNN. In some embodiments,one or more of the first model and the second model may include aConvolutional Neural Network architecture (CNN) for text processingfollowed by a regression model, and wherein the regression model outputsthe estimated customer satisfaction score based on an output of the CNNand the metadata of the given interaction.

Further embodiments of the invention include systems and methods ofcalculating a score for a chain of interactions in a call center,performed on a computer having a processor, memory, and one or more codesets stored in the memory and executed by the processor, the systems andmethods including: during a first training phase, training a first modelwhich, given an interaction and associated interaction metadata,predicts an initial estimated customer satisfaction score for the giveninteraction; during a second training phase, training a second modelwhich, given an interaction in a given chain of interactions, animmediately preceding interaction in the given chain of interactions,associated interaction metadata of the interaction and the immediatelypreceding interaction, and one or more features of the given chain ofinteractions, predicts a refined estimated customer satisfaction scorefor the given interaction; and during an inference phase: given a chainof interactions and associated metadata of each interaction, computingan initial estimated customer satisfaction score for each interaction inthe chain of interactions using the first model; beginning with a secondinteraction in the given chain of interactions and associated metadataof each interaction, computing a refined estimated customer satisfactionscore for each interaction in the chain of interactions using the secondmodel; combining the refined estimated customer satisfaction score ofeach interaction into a combined customer satisfaction score for thegiven chain of interactions; and outputting the combined customersatisfaction score.

These and other aspects, features and advantages will be understood withreference to the following description of certain embodiments of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanied drawings. Embodiments of the invention areillustrated by way of example and not limitation in the figures of theaccompanying drawings, in which like reference numerals indicatecorresponding, analogous or similar elements, and in which:

FIG. 1 shows a high-level diagram illustrating an exemplaryconfiguration of a system 100 for performing one or more aspects of theinvention;

FIG. 2 is a high-level diagram illustrating an example configuration ofa method workflow for calculating a score for a chain of interactions ina call center according to at least one embodiment of the invention;

FIG. 3 shows a method of training a first model according to at leastone embodiment of the invention;

FIG. 4 shows an example regression model according to at least oneembodiment of the invention;

FIG. 5 shows an example regression model according to at least oneembodiment of the invention;

FIG. 6 shows a method of training a second model according to at leastone embodiment of the invention;

FIG. 7 shows a method of determining a customer experience score for achain of interactions according to at least one embodiment of theinvention; and

FIG. 8 shows an example screenshot of a user interface (UI) of anapplication on which embodiments of the invention have been implemented.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn accuratelyor to scale. For example, the dimensions of some of the elements may beexaggerated relative to other elements for clarity, or several physicalcomponents may be included in one functional block or element. Further,where considered appropriate, reference numerals may be repeated amongthe figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Although embodiments of the invention are not limited in this regard,discussions utilizing terms such as, for example, “processing,”“computing,” “calculating,” “determining,” “establishing”, “analyzing”,“checking”, or the like, may refer to operation(s) and/or process(es) ofa computer, a computing platform, a computing system, or otherelectronic computing device, that manipulates and/or transforms datarepresented as physical (e.g., electronic) quantities within thecomputer's registers and/or memories into other data similarlyrepresented as physical quantities within the computer's registersand/or memories or other information non-transitory processor-readablestorage medium that may store instructions, which when executed by theprocessor, cause the processor to perform operations and/or processes.Although embodiments of the invention are not limited in this regard,the terms “plurality” and “a plurality” as used herein may include, forexample, “multiple” or “two or more”. The terms “plurality” or “aplurality” may be used throughout the specification to describe two ormore components, devices, elements, units, parameters, or the like. Theterm set when used herein may include one or more items. Unlessexplicitly stated, the method embodiments described herein are notconstrained to a particular order or sequence. Additionally, some of thedescribed method embodiments or elements thereof may occur or beperformed simultaneously, at the same point in time, or concurrently.

Embodiments of the invention may provide systems and methods which maycompute a customer experience score for a chain of interactions (e.g.,calls, chats, and/or emails, etc.). Embodiments of the invention mayimplement one or more algorithms which, in a first training phase, mayuse customer feedback such as, e.g., survey data and/or information, totrain a deep learning based model that predicts a customer experiencemetric relating to an interaction in a chain of interactions. Then, in asecond training phase, one or more algorithms may be implemented toenhance or refine the model by adding or otherwise combining it with apredicted customer experience metric of the previous interaction in thechain of interactions. Finally, in an inference phase, one or morealgorithms may be implemented to generate or otherwise calculate asingle customer satisfaction score for a given chain of interactions,e.g., based on the first and second models.

In some embodiments, the first model training phase may relate initiallyto a single interaction in an interaction chain. Embodiments of theinvention may perform one or more of the following steps: (1) collectinteractions for which customer (e.g., interactant) survey informationis available; (2) transcribe the calls using, e.g., available automaticspeech recognition (ASR) software or other speech-to-text (STT)software; (3) preprocess the text (e.g., remove function words, etc.);(4) for each interaction, extract one or more metadata features, forexample: duration (sec), non-speech percentage, weekend (0/1), day inweek, hour, number of times the customer was placed on hold, number oftimes the customer was transferred, number of times the customer wasconferenced, hold duration percentage, time of day (e.g., morning,afternoon, evening, night, etc.), finance type, credit class code, etc.;and (5) apply a deep learning algorithm, e.g., a Convolutional NeuralNetwork (CNN), with a unique architecture to support both structured andunstructured features to learn a customer satisfaction score from aninteraction's text, metadata and survey results, in order to predict aninitial estimated customer satisfaction score for a given interaction.

In some embodiments, the second model training phase may relate to asingle interaction in an interaction chain. Embodiments of the inventionmay perform one or more of the following steps: (1) select or otherwiseidentify interaction chains; (2) for each interaction in the chain,predict an initial customer experience score using the model trained inthe previous phase; (3) in addition to features used in the first modeltraining phase, for each interaction add features from the previousinteraction in the chain as follows: (4) add the estimated customerexperience score of the previous interaction; (5) add metadata of theprevious interaction; (6) for each interaction, add chain features, suchas, for example, place in chain order, elapsed hours from the previousinteraction, etc.; and (7) re-train the model using, for example, a CNNor similar algorithm, in order to predict a refined estimated customersatisfaction score for a given interaction.

During an inference phase, embodiments of the invention may perform oneor more of the following steps: (1) given a chain of interactions, applyASR to every interaction in the chain; (2) use the first phase model toestimate a customer experience score for every interaction in the chain;(3) use the second phase model to refine the customer experience scorestarting from the second interaction in the chain; (4) and combine allscores to a single score per chain as follows:

${{CXscore}({chain})} = {\frac{1 - r}{1 - r^{n}}{\sum\limits_{i = 1}^{n}\left( {{{CXscore}(i)}*r^{i - 1}} \right)}}$where n is the number of interactions in the chain and r is a weightpower factor.

These and other features of embodiments of the invention will be furtherunderstood with reference to FIGS. 1-8 as described herein.

FIG. 1 shows a high-level diagram illustrating an exemplaryconfiguration of a system 100 for performing one or more aspects of theinvention described herein, according to at least one embodiment of theinvention. System 100 includes network 105, which may include theInternet, one or more telephony networks, one or more network segmentsincluding local area networks (LAN) and wide area networks (WAN), one ormore wireless networks, or a combination thereof. System 100 alsoincludes a system server 110 constructed in accordance with one or moreembodiments of the invention. In some embodiments, system server 110 maybe a stand-alone computer system. In other embodiments, system server110 may include a network of operatively connected computing devices,which communicate over network 105. Therefore, system server 110 mayinclude multiple other processing machines such as computers, and morespecifically, stationary devices, mobile devices, terminals, and/orcomputer servers (collectively, “computing devices”). Communication withthese computing devices may be, for example, direct or indirect throughfurther machines that are accessible to the network 105.

System server 110 may be any suitable computing device and/or dataprocessing apparatus capable of communicating with computing devices,other remote devices or computing networks, receiving, transmitting andstoring electronic information and processing requests as furtherdescribed herein. System server 110 is, therefore, intended to representvarious forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers and/or networked orcloud-based computing systems capable of employing the systems andmethods described herein.

System server 110 may include a server processor 115 which isoperatively connected to various hardware and software components thatserve to enable operation of the system 100. Server processor 115 servesto execute instructions to perform various operations relating toadvanced search, and other functions of embodiments of the invention asdescribed in greater detail herein. Server processor 115 may be one orseveral processors, a central processing unit (CPU), a graphicsprocessing unit (GPU), a multi-processor core, or any other type ofprocessor, depending on the particular implementation.

System server 110 may be configured to communicate via communicationinterface 120 with various other devices connected to network 105. Forexample, communication interface 120 may include but is not limited to,a modem, a Network Interface Card (NIC), an integrated networkinterface, a radio frequency transmitter/receiver (e.g., Bluetoothwireless connection, cellular, Near-Field Communication (NFC) protocol,a satellite communication transmitter/receiver, an infrared port, a USBconnection, and/or any other such interfaces for connecting the systemserver 110 to other computing devices and/or communication networks suchas private networks and the Internet.

In certain implementations, a server memory 125 is accessible by serverprocessor 115, thereby enabling server processor 115 to receive andexecute instructions such as a code, stored in the memory and/or storagein the form of one or more software modules 130, each modulerepresenting one or more code sets. The software modules 130 may includeone or more software programs or applications (collectively referred toas the “server application”) having computer program code or a set ofinstructions executed partially or entirely in server processor 115 forcarrying out operations for aspects of the systems and methods disclosedherein, and may be written in any combination of one or more programminglanguages. Server processor 115 may be configured to carry outembodiments of the present invention by, for example, executing code orsoftware, and may execute the functionality of the modules as describedherein.

In accordance with embodiments of FIG. 1 , the exemplary softwaremodules may include a communication module and other modules asdescribed here. The communication module may be executed by serverprocessor 115 to facilitate communication between system server 110 andthe various software and hardware components of system 100, such as, forexample, server database 135, client device 140, and/or externaldatabase 175 as described herein.

Of course, in some embodiments, server modules 130 may include more orless actual modules which may be executed to enable these and otherfunctionalities of the invention. The modules described herein are,therefore, intended to be representative of the various functionalitiesof system server 110 in accordance with some embodiments of theinvention. It should be noted that, in accordance with variousembodiments of the invention, server modules 130 may be executedentirely on system server 110 as a stand-alone software package, partlyon system server 110 and partly on user device 140, or entirely on userdevice 140.

Server memory 125 may be, for example, random access memory (RAM) or anyother suitable volatile or non-volatile computer readable storagemedium. Server memory 125 may also include storage which may takevarious forms, depending on the particular implementation. For example,the storage may contain one or more components or devices such as a harddrive, a flash memory, a rewritable optical disk, a rewritable magnetictape, or some combination of the above. In addition, the memory and/orstorage may be fixed or removable. In addition, memory and/or storagemay be local to the system server 110 or located remotely.

In accordance with further embodiments of the invention, system server110 may be connected to one or more database(s) 135, for example,directly or remotely via network 105. Database 135 may include any ofthe memory configurations as described herein, and may be in direct orindirect communication with system server 110. In some embodiments,database 135 may store information relating to user documents. In someembodiments, database 135 may store information related to one or moreaspects of the invention.

As described herein, among the computing devices on or connected to thenetwork 105 may be one or more user devices 140. User device 10 may beany standard computing device. As understood herein, in accordance withone or more embodiments, a computing device may be a stationarycomputing device, such as a desktop computer, kiosk and/or othermachine, each of which generally has one or more processors, such asuser processor 145, configured to execute code to implement a variety offunctions, a computer-readable memory, such as user memory 155, a usercommunication interface 150, for connecting to the network 105, one ormore user modules, such as user module 160, one or more input devices,such as input devices 165, and one or more output devices, such asoutput devices 170. Typical input devices, such as, for example, inputdevices 165, may include a keyboard, pointing device (e.g., mouse ordigitized stylus), a web-camera, and/or a touch-sensitive display, etc.Typical output devices, such as, for example output device 170 mayinclude one or more of a monitor, display, speaker, printer, etc.

In some embodiments, user module 160 may be executed by user processor145 to provide the various functionalities of user device 140. Inparticular, in some embodiments, user module 160 may provide a userinterface with which a user of user device 140 may interact, to, amongother things, communicate with system server 110.

Additionally or alternatively, a computing device may be a mobileelectronic device (“MED”), which is generally understood in the art ashaving hardware components as in the stationary device described above,and being capable of embodying the systems and/or methods describedherein, but which may further include componentry such as wirelesscommunications circuitry, gyroscopes, inertia detection circuits,geolocation circuitry, touch sensitivity, among other sensors.Non-limiting examples of typical MEDs are smartphones, personal digitalassistants, tablet computers, and the like, which may communicate overcellular and/or Wi-Fi networks or using a Bluetooth or othercommunication protocol. Typical input devices associated withconventional MEDs include, keyboards, microphones, accelerometers, touchscreens, light meters, digital cameras, and the input jacks that enableattachment of further devices, etc.

In some embodiments, user device 140 may be a “dummy” terminal, by whichprocessing and computing may be performed on system server 110, andinformation may then be provided to user device 140 via servercommunication interface 120 for display and/or basic data manipulation.In some embodiments, modules depicted as existing on and/or executing onone device may additionally or alternatively exist on and/or execute onanother device. For example, in some embodiments, one or more modules ofserver module 130, which is depicted in FIG. 1 as existing and executingon system server 110, may additionally or alternatively exist and/orexecute on user device 140. Likewise, in some embodiments, one or moremodules of user module 160, which is depicted in FIG. 1 as existing andexecuting on user device 140, may additionally or alternatively existand/or execute on system server 110.

FIG. 2 is a high-level diagram illustrating an example configuration ofa method workflow 200 for calculating a score for a chain ofinteractions in a call center according to at least one embodiment ofthe invention. As described herein, embodiments of systems and methodsof the invention enable computation of a customer experience score for achain of interactions (e.g., calls, chats, emails, etc.), focusing on achain of interactions and using interaction content and explicitfeedback from the customer. Embodiments of the invention include atleast three main phases, as shown in FIG. 2 : a first training phase(also referred to herein as “Training Phase 1”), a second training phase(also referred to herein as “Training Phase 2”), and an Inference phase.Furthermore, as described herein, in some embodiments, one or more ofthese phases may be implemented iteratively. For example, Training Phase1 and Training Phase 2 may be implemented iteratively, e.g., to furtherrefine the various training models described herein, and/or may receivefeedback from the Inference Phase to further inform one or more of themodels.

In some embodiments, method workflow 200 may be performed on a computer(e.g., system server 110) having a processor (e.g., server processor115), memory (e.g., server memory 125), and one or more code sets orsoftware (e.g., server module(s) 130) stored in the memory and executingin or executed by the processor. Method workflow 200 begins with thefirst of the three processes, at step 205, when, during a first trainingphase, the processor is configured to train a first model, as describedin detail with reference to FIG. 3 .

Turning to FIG. 3 , a method 300 of training a first model is describedaccording to at least one embodiment of the invention. At step 305, insome embodiments, the processor may collect a plurality of interactionsin which each of the plurality of interactions includes associatedsurvey information. In some embodiments, an interaction may be a voicerecording, an e-mail, a messaging chat, a text message, etc. In someembodiments, the processor may receive as an input to the algorithm adataset of e-mails, a dataset of chats, and/or a dataset of voiceinteractions/conversations (e.g., previously converted to text). At step210, in some embodiments, for example when voice or any other non-textinteractions have not been previously converted to text), the processormay convert any non-text interactions (e.g., voice conversations) of theplurality of interactions into text, using, for example, ASR software.In some embodiments, the processor may further receive as an inputautomatically transcribed content (ASR) of a dataset of voiceconversations.

In some embodiments, all input interactions may have a survey scoreavailable in a database. For example, a customer may have been contactedto provide survey information after an interaction or requested toprovide survey information at the conclusion of an interaction (e.g.,asking to rate an interaction on a scale from 1 to 5, etc.). Such surveydata may be stored in a database and accessed by the processor whenneeded, as described herein.

In some embodiments, at step 315, the processor may receive as inputmetadata features describing each of the voice conversations and/orother interactions in a dataset. In some embodiments, the processor mayextract the metadata from the voice conversations and/or otherinteractions. A non-exhaustive list of metadata features may include(but are not limited to) the following provided in Table 1 below:

TABLE 1 Feature group Feature name Description Interaction Duration(sec) Call duration features Non speech percentage Silence as a portionof a call (voice) Was the call made on Weekend (0/1) weekend? Day inweek What day in a week was the call made [1-7] Hour What hour? #Holds(voice) Number of holds in a call #Transfers (voice) Number of transfersin a call #Conference (voice) Number of conferences in a call Holdduration Hold time as a portion of a percentage (voice) call Morning,afternoon, Was the call made in the evening, night morning, afternoon... Customer Finance type Customer's finance type features Credit classcode Customer's credit class Agent Skill group Agent's skill groupfeatures Call amount Number of calls made by the agent in a period oftime Agent group Survey CSAT score A score between 0 and 10 given by thecustomer after the interaction

In some embodiments, at step 320, the processor may be configured topreprocess the text of each of the plurality of interactions. Forexample, in some embodiments, the processor may take an input (e.g., asa string) the text of a given interaction and use pre-trained in-domainembeddings to transform words and phrases from the input text into oneor more vectors. In some embodiments, the processor may train one ormore word embeddings using existing tools such as: word2vec, GloVe,fastText, GenSim and others. Word embedding, as understood herein, isthe collective name for a set of language modeling and feature learningtechniques in natural language processing (NLP) in which words orphrases from a given vocabulary are mapped to vectors of real numbers.Conceptually this involves a mathematical embedding from a space withone dimension per word to a continuous vector space with a much lowerdimension. In some embodiments, one or more of these NLP techniques maybe implemented in order to create the word embeddings, which may beimplemented in later steps of method 300, as described herein. In someembodiments, the output may be word embeddings of the typeVector<float>.

In some embodiments, at step 325, the processor may be configured totrain a first model which, given a processed text of a given interactionand metadata associated with the given interaction, predicts an initialestimated customer satisfaction score for the given interaction. In someembodiments, the processor may train a regressor that given a processedtext of the interaction and its meta-data predicts a customersatisfaction (customer experience) score. Those of skill in the art willunderstand that a variety of machine learning based regression modelsmay be used according to embodiments of the invention. Two such examplesare provided herein.

Convolutional Neural Network (CNN) with an auxiliary output:

Turning briefly to FIG. 4 , regression model 400 depicted includes afairly standard CNN architecture for addressing the text features aswell as a more simple Deep Neural Network (DNN) for the meta-features.In some embodiments, the predictions of both networks may beconcatenated and fed to a linear regression output function. Anauxiliary output may be used to load-balance the learning processbetween the two networks.

Convolutional Neural Network followed by a classic regression model:

Turning briefly to FIG. 5 , regression model 500 shows a tandem of a CNNmodel for the text features followed by a classic regression model.Examples of classic regression models which may be implemented includeSVM, Random Forest, XGBoost and Gradient Boosting.

Returning to FIG. 3 , in some embodiments, at step 325, the processormay be configured to train the first model based on the text, metadata,and survey information of each of the plurality of interactions. Theoutput of this phase is a regression model that predicts the customerexperience score based on the content of the interaction and itsmetadata features.

Returning to FIG. 2 , Method workflow 200 continues with the second ofthe three processes, at step 210, when, during a second training phase,the processor is configured to train a second model, as described indetail with reference to FIG. 6 .

Turning to FIG. 6 , a method 600 of training a second model is describedaccording to at least one embodiment of the invention. At step 605A insome embodiments, the processor may collect or otherwise identify theplurality of interactions used in the first phase (detailed in FIG. 3 ).Likewise, at step 605B, for each interaction used in the first phase,the processor may be configured to collect or otherwise identify theprevious interaction made by the same customer. In some embodiments, aprevious interaction may be from a different channel as the currentinteraction. For example, the current interaction may be a phone call,while the previous interaction may be a chat. In some embodiments, incase there is no previous interaction, predefined default values may beused for the predicted customer experience score and the chain featuresthat would have been provided in connection with a previous interaction.

In some embodiments, an interaction may be a voice recording, an e-mail,a messaging chat, a text message, etc. In some embodiments, theprocessor may receive as an input to the algorithm a dataset of e-mails,a dataset of chats, and/or a dataset of voice interactions/conversations(e.g., previously converted to text). At steps 610A and 610Brespectively, in some embodiments (for example, when voice or any othernon-text interactions have not been previously converted to text), theprocessor may convert any non-text interactions (e.g., voiceconversations) of the plurality of interactions and prior interactionsinto text, using, for example, ASR software. In some embodiments, theprocessor may further receive as an input automatically transcribedcontent (ASR) of a dataset of voice conversations.

As discussed with reference to FIG. 3 , in some embodiments, all inputinteractions may have a survey score available in a database. Forexample, a customer may have been contacted to provide surveyinformation after an interaction or requested to provide surveyinformation at the conclusion of an interaction (e.g., asking to rate aninteraction on a scale from 1 to 5, etc.). Such survey data may bestored in a database and accessed by the processor when needed, asdescribed herein. In some embodiments, previous interactions may notnecessarily have or require survey data.

In some embodiments, at steps 615A and 615B respectively, the processormay receive as input metadata features describing each of the voiceconversations and/or other interactions in a dataset. In someembodiments, the processor may extract the metadata from the voiceconversations and/or other interactions. A non-exhaustive list ofmetadata features may include (but are not limited to) the itemsprovided in Table 1 above.

In some embodiments, at steps 620A and 620B respectively, the processormay be configured to preprocess the text of each of the plurality ofinteractions and previous interactions.

For example, as described with regard to step 320 of FIG. 3 , in someembodiments, the processor may take an input (e.g., as a string) thetext of a given interaction and use pre-trained in-domain embeddings totransform words and phrases from the input text into one or morevectors. In some embodiments, the processor may train one or more wordembeddings using existing tools such as: word2vec, GloVe, fastText,GenSim and others. In some embodiments, one or more of these NLPtechniques may be implemented in order to create the word embeddings,which may be implemented in later steps of method 600, as describedherein. In some embodiments, the output may be word embeddings of thetype Vector<float>.

In some embodiments, at steps 625A and 625B respectively, the processormay be configured to apply the first model (described in detail withregard to FIG. 3 ), which, given a processed text of a given interactionand metadata associated with the given interaction, predicts an initialestimated customer satisfaction score for the given interaction. In someembodiments, at steps 630A and 630B respectively, the processor may beconfigured to generate an interim estimated customer satisfaction scorefor each of the plurality of interactions and immediately precedinginteractions.

In some embodiments, at step 635, the processor may be configured totrain a second model based on one or more features of the chain ofinteractions, and the interim estimated customer satisfaction score foreach of the plurality of interactions and immediately precedinginteractions. In some embodiments, the second model may receive as aninput the predicted customer experience score (interim estimatedcustomer satisfaction score) of the current interaction (e.g., a realnumber between 0 and 10) and the predicted customer experience score(interim estimated customer satisfaction score) of the previousinteraction (e.g., a real number between 0 and 10). In some embodiments,if a previous interaction is not available, an average survey score maybe substituted in the training dataset.

In some embodiments, a non-exhaustive list of chain features may include(but are not limited to) the following provided in Table 2 below:

TABLE 2 Feature name Description Default Place in chain First call,second, ... 1 Elapsed hours Hours between current and 4320 Previous theprevious call (half a year) channel The channel of the Not availableprevious interaction

In some embodiments, a machine learning based regression algorithm maybe implemented, such as SVM, random forest, XGBoost and gradientboosting. The output is a second training model which, given a processedtext of a given interaction and metadata associated with the giveninteraction, processed text and associated metadata of an immediatelypreceding interaction in a given chain of interactions, and one or morefeatures of the chain of interactions, predicts a refined estimatedcustomer satisfaction score for the given interaction.

Returning to FIG. 2 , Method workflow 200 continues with the third ofthe three processes, at step 215, when, during an inference phase, theprocessor is configured, to generate or otherwise calculate a singlecustomer satisfaction score for a given chain of interactions, e.g.,based on the first and second models, as described in detail withreference to FIG. 7 .

Turning to FIG. 7 , a method 700 of determining a customer experience(customer satisfaction) score for a chain of interactions is describedaccording to at least one embodiment of the invention. At step 705, insome embodiments, the processor may input a chain of interactions(detailed in FIG. 3 ). In some embodiments, at step 710, in the eventone of the interactions in the chain of interactions is a non-textinteraction (e.g., a voice interaction such as a phone call), theprocessor may be configured to automatically transcribe or otherwiseconvert the non-text interaction to text (e.g., using ASR). At step 715,in some embodiments, the processor may be configured to extract themetadata from the voice conversations and/or other interactions of thechain of interactions. A non-exhaustive list of metadata features mayinclude (but are not limited to) the items provided in Table 1 above. Itshould be noted that, in some embodiments, no survey information isrequired for the received chain of interactions (interaction chain).

In some embodiments, at step 720, the processor may be configured topreprocess the text of each of the plurality of interactions in thechain of interactions, for example, as described with regard to step 320of FIG. 3 .

In some embodiments, at step 725, the processor may be configured toapply the first model to each interaction in the interaction chain andcompute an initial estimated customer satisfaction score (customerexperience score) for each interaction in the chain of interactions,using the first model. Next, at step 730, in some embodiments, beginningwith the second interaction in the chain of interactions and associatedmetadata of each interaction, computing a refined estimated customersatisfaction score for each interaction in the chain of interactions,using the second model. That is, starting with the second interaction,the processor may be configured to apply the phase 2 model on model 1prediction of the customer experience score of the current and theprevious interactions, incorporating the additional chain data, asdescribed herein, to compute a refined estimated customer satisfactionscore for each interaction in the interaction chain.

Finally, in some embodiments, at step 735, the processor may beconfigured to merge or otherwise combine the refined estimated customersatisfaction score of each interaction into a combined customersatisfaction score for the given chain of interactions, and output thecombined customer satisfaction score, e.g., to a display screen. In someembodiments, the processor may combine customer experience (CX) scoresof each interaction, to determine one score per chain as follows:

${{CXscore}({chain})} = {\frac{1 - r}{1 - r^{n}}{\sum\limits_{i = 1}^{n}\left( {{{CXscore}(i)}*r^{i - 1}} \right)}}$where n is the number of the interaction in the chain and r is apredetermined weight power factor.

For example, suppose a chain has three (3) interactions: two consecutivephone calls followed by an e-mail. After applying phase 1 and phase 2models, the CX scores are: 7, 4 and 2. Assuming r=2, thenCXscore=(1/7)*7+(2/7)*4+(4/7)*2=23/7=3.3. Alternatively, in otherembodiments other formulae that combine the weights of individualinteractions may also be implemented to determine the customerinteraction score for the chain of interactions. FIG. 8 shows an examplescreenshot 800 of a user interface (UI) of an application on whichembodiments of the invention have been implemented. As can be seen inscreenshot 800, both scores of individual interactions and of theinteraction chain are displayed.

Embodiments of the invention provide significant improvements overcurrently available systems and methods. For example, embodiments of theinvention enable users to identify patterns that lead to bad interactionchains, drill down to the reason for bad interaction chains, rankbusiness processes based on the customer experience score of theinteraction chains, etc.

Unless explicitly stated, the method embodiments described herein arenot constrained to a particular order or sequence. Furthermore, allformulas described herein are intended as examples only and other ordifferent formulas may be used. Additionally, some of the describedmethod embodiments or elements thereof may occur or be performed at thesame point in time.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents may occur to those skilled in the art. It is, therefore, tobe understood that the appended claims are intended to cover all suchmodifications and changes as fall within the true spirit of theinvention.

Various embodiments have been presented. Each of these embodiments mayof course include features from other embodiments presented, andembodiments not specifically described may include various featuresdescribed herein.

What is claimed is:
 1. A method of calculating a score for a chain ofinteractions in a call center, performed on a computer having aprocessor, memory, and one or more code sets stored in the memory andexecuted by the processor, the method comprising: during a firsttraining phase, training a first model, wherein training the first modeluses: text of a given interaction processed into vectors, surveyinformation, and metadata associated with the given interaction, thetraining such that the first model predicts an initial estimatedcustomer satisfaction score for the given interaction; during a secondtraining phase, training a second model, wherein training the secondmodel uses: text of a given interaction and metadata associated with thegiven interaction, processed text and associated metadata of animmediately preceding interaction in a given chain of interactions, andone or more features of the chain of interactions, and the initialestimated customer satisfaction score for the immediately precedinginteraction, the score produced by the first model, the training suchthat the second model predicts a refined estimated customer satisfactionscore for the given interaction; and during an inference phase: given achain of interactions and associated metadata of each interaction,computing the initial estimated customer satisfaction score for eachinteraction in the chain of interactions using the first model;beginning with a second interaction in the given chain of interactionsand associated metadata of each interaction, computing a refinedestimated customer satisfaction score for each interaction in the chainof interactions using the second model; combining the refined estimatedcustomer satisfaction score of each interaction into a combined customersatisfaction score for the given chain of interactions; and outputtingthe combined customer satisfaction score.
 2. The method as in claim 1,wherein the first training phase comprises: collecting by the processora plurality of interactions, wherein each of the plurality ofinteractions comprises associated survey information; converting by theprocessor any non-text interactions of the plurality of interactionsinto text; preprocessing by the processor text of each of the pluralityof interactions; extracting by the processor a plurality of metadatafeatures from each of the plurality of interactions; and training by theprocessor the first model based on the text, metadata, and surveyinformation of each of the plurality of interactions.
 3. The method asin claim 2, wherein the second training phase comprises: for the sameplurality of interactions, as well as an immediately precedinginteraction of each of the plurality of interactions in a given chain ofinteractions: converting by the processor any non-text interactions intotext; preprocessing by the processor text of each of the plurality ofinteractions and immediately preceding interactions; extracting by theprocessor a plurality of metadata features from each of the plurality ofinteractions and immediately preceding interactions; applying by theprocessor the first model to each of the plurality of interactions andimmediately preceding interactions; generating by the processor aninterim estimated customer satisfaction score for each of the pluralityof interactions and immediately preceding interactions; and training bythe processor the second model based on one or more features of thechain of interactions, and the interim estimated customer satisfactionscore for each of the plurality of interactions and immediatelypreceding interactions.
 4. The method as in claim 1, wherein aninteraction comprises at least one of a voice recording, an e-mail, anda messaging chat.
 5. The method as in claim 1, wherein the associatedsurvey information comprises a survey score provided by a customer whowas a party to the given interaction.
 6. The method as in claim 1,wherein preprocessing comprises transforming by the processor at leastone of one or more words and one or more phrases from the text of agiven interaction into one or more word embedding vectors, using one ormore pre-trained in-domain embeddings.
 7. The method as in claim 1,wherein one or more of the first model and the second model comprises aConvolutional Neural Network architecture (CNN) and a Deep NeuralNetwork architecture (DNN), and wherein the CNN and the DNN areload-balanced with an auxiliary output of the CNN.
 8. The method as inclaim 1, wherein one or more of the first model and the second modelcomprises a Convolutional Neural Network architecture (CNN) for textprocessing followed by a regression model, and wherein the regressionmodel outputs the estimated customer satisfaction score based on anoutput of the CNN and the metadata of the given interaction.
 9. A systemof calculating a score for a chain of interactions in a call center,comprising: a computer having a processor and memory, and one or morecode sets stored in the memory and executed by the processor, whichconfigure the processor to: during a first training phase, train a firstmodel, wherein training the first model uses: text of a giveninteraction processed into vectors, survey information, and metadataassociated with the given interaction, the training such that the firstmodel predicts an initial estimated customer satisfaction score for thegiven interaction; during a second training phase, train a second model,wherein training the second model uses: text of a given interaction andmetadata associated with the given interaction, processed text andassociated metadata of an immediately preceding interaction in a givenchain of interactions, and one or more features of the chain ofinteractions, and the initial estimated customer satisfaction score forthe immediately preceding interaction, the score produced by the firstmodel, the training such that the second model predicts a refinedestimated customer satisfaction score for the given interaction; andduring an inference phase: given a chain of interactions and associatedmetadata of each interaction, compute an initial estimated customersatisfaction score for each interaction in the chain of interactionsusing the first model; beginning with a second interaction in the givenchain of interactions and associated metadata of each interaction,compute a refined estimated customer satisfaction score for eachinteraction in the chain of interactions using the second model; combinethe refined estimated customer satisfaction score of each interactioninto a combined customer satisfaction score for the given chain ofinteractions; and output the combined customer satisfaction score. 10.The system as in claim 9, wherein, during the first training phase, theprocessor is configured to: collect a plurality of interactions, whereineach of the plurality of interactions comprises associated surveyinformation; convert any non-text interactions of the plurality ofinteractions into text; preprocess text of each of the plurality ofinteractions; extract a plurality of metadata features from each of theplurality of interactions; and train the first model based on the text,metadata, and survey information of each of the plurality ofinteractions.
 11. The system as in claim 10, wherein, during the secondtraining phase, the processor is configured to: for the same pluralityof interactions, as well as an immediately preceding interaction of eachof the plurality of interactions in a given chain of interactions:convert any non-text interactions into text; preprocess text of each ofthe plurality of interactions and immediately preceding interactions;extract a plurality of metadata features from each of the plurality ofinteractions and immediately preceding interactions; apply the firstmodel to each of the plurality of interactions and immediately precedinginteractions; generate an interim estimated customer satisfaction scorefor each of the plurality of interactions and immediately precedinginteractions; and train the second model based on one or more featuresof the chain of interactions, and the interim estimated customersatisfaction score for each of the plurality of interactions andimmediately preceding interactions.
 12. The system as in claim 9,wherein an interaction comprises at least one of a voice recording, ane-mail, and a messaging chat.
 13. The system as in claim 9, wherein theassociated survey information comprises a survey score provided by acustomer who was a party to the given interaction.
 14. The system as inclaim 9, wherein preprocessing comprises transforming by the processorat least one of one or more words and one or more phrases from the textof a given interaction into one or more word embedding vectors, usingone or more pre-trained in-domain embeddings.
 15. The system as in claim9, wherein one or more of the first model and the second model comprisesa Convolutional Neural Network architecture (CNN) and a Deep NeuralNetwork architecture (DNN), and wherein the CNN and the DNN areload-balanced with an auxiliary output of the CNN.
 16. The system as inclaim 9, wherein one or more of the first model and the second modelcomprises a Convolutional Neural Network architecture (CNN) for textprocessing followed by a regression model, and wherein the regressionmodel outputs the estimated customer satisfaction score based on anoutput of the CNN and the metadata of the given interaction.
 17. Amethod of calculating a score for a chain of interactions in a callcenter, performed on a computer having a processor, memory, and one ormore code sets stored in the memory and executed by the processor, themethod comprising: during a first training phase, training a firstmodel, wherein training the first model uses: text of a giveninteraction processed into vectors, survey information, and associatedinteraction metadata, the training such that the first model predicts aninitial estimated customer satisfaction score for the given interaction;during a second training phase, training a second model, whereintraining the second model uses: text of a given interaction in a givenchain of interactions of an immediately preceding interaction in thegiven chain of interactions, associated interaction metadata of theinteraction and the immediately preceding interaction, and one or morefeatures of the given chain of interactions, and the initial estimatedcustomer satisfaction score for the immediately preceding interaction,the score produced by the first model, the training such that the secondmodel predicts a refined estimated customer satisfaction score for thegiven interaction; and during an inference phase: given a chain ofinteractions and associated metadata of each interaction, computing aninitial estimated customer satisfaction score for each interaction inthe chain of interactions using the first model; beginning with a secondinteraction in the given chain of interactions and associated metadataof each interaction, computing a refined estimated customer satisfactionscore for each interaction in the chain of interactions using the secondmodel; combining the refined estimated customer satisfaction score ofeach interaction into a combined customer satisfaction score for thegiven chain of interactions; and outputting the combined customersatisfaction score.
 18. The method as in claim 17, wherein the firsttraining phase comprises: collecting by the processor a plurality ofinteractions, wherein each of the plurality of interactions comprisesassociated survey information; converting by the processor any non-textinteractions of the plurality of interactions into text; preprocessingby the processor text of each of the plurality of interactions;extracting by the processor a plurality of metadata features from eachof the plurality of interactions; and training by the processor thefirst model based on the text, metadata, and survey information of eachof the plurality of interactions.
 19. The method as in claim 18, whereinthe second training phase comprises: for the same plurality ofinteractions, as well as an immediately preceding interaction of each ofthe plurality of interactions in a given chain of interactions:converting by the processor any non-text interactions into text;preprocessing by the processor text of each of the plurality ofinteractions and immediately preceding interactions; extracting by theprocessor a plurality of metadata features from each of the plurality ofinteractions and immediately preceding interactions; applying by theprocessor the first model to each of the plurality of interactions andimmediately preceding interactions; generating by the processor aninterim estimated customer satisfaction score for each of the pluralityof interactions and immediately preceding interactions; and training bythe processor the second model based on one or more features of thechain of interactions, and the interim estimated customer satisfactionscore for each of the plurality of interactions and immediatelypreceding interactions.
 20. The method as in claim 17, wherein aninteraction comprises at least one of a voice recording, an e-mail, anda messaging chat.